Several interactive displays are known in the prior art. For example, a user interface platform was developed in the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST October 1997:14-17. This article describes how the metaDESK includes a near-horizontal graphical surface that is used to display two-dimensional (2D) geographical information. A computer vision system inside the desk unit (i.e., below the graphical surface) includes infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface to provide images that are visible to a user from above the graphical display surface. The article further teaches that the IR camera can detect a distinctive pattern provided on the undersurface of passive objects called “phicons” that are placed on the graphical surface. Thus, the IR camera detects an IR pattern (which is transparent to visible light) applied to the bottom of a “Great Dome phicon” and responds by displaying a map of the MIT campus on the graphical surface, with the actual location of the Great Dome in the map positioned where the Great Dome phicon is located. Moving the Great Dome phicon over the graphical surface manipulates the displayed map by rotating or translating the map in correspondence to the movement of the phicon by a user. Clearly, the IR vision-sensing system used in this prior art interactive display is able to detect objects like the phicon, based upon the pattern applied to it. There is no discussion of details involved in simply detecting an object without a pattern, or in determining a relative position of an object above the display surface.
A similar technique for sensing objects on a display surface is disclosed in several papers published by Jun Rekimoto of Sony Computer Science Laboratory, Inc., in collaboration with others. These papers briefly describe a “HoloWall” and a “HoloTable,” both of which use IR light to detect objects that are proximate to or in contact with a display panel on which a rear-projected image is visible. The rear-projection panel, which is vertical in the HoloWall and horizontal in the HoloTable, is semi-opaque and diffusive, so that objects reflecting IR light back through the panel become more clearly visible to an IR camera as they approach and then contact the panel. The objects thus detected can be a user's fingers or hand, or other objects. Again, these papers are generally silent regarding the process used for detecting an object based upon the IR light reflected from the object and also fail to discuss determining the relative separation between an object and the display surface.
Clearly, it is known in the art to employ reflected IR light to detect an object placed on a diffusing display surface. The present invention also relates to sensing objects on the display surface of a novel interactive display system that is similar in some ways to the prior art interactive display systems discussed above. This new interactive display table employs a computer vision-based sensing system that produces a signal corresponding to an image of the display surface that indicates where objects that reflect IR light are placed on the display surface. In particular, the pixel intensity at each pixel location in this image indicates whether there is any IR reflective material at that location on the table, but it is still necessary to determine the precise location of an object on the physical display surface, particularly, since an object can appear in the image as a plurality of objects, particularly if the object is not fully in contact with the diffusing display surface. The actual number of objects and their location on the display surface are important to enable software applications to reason about objects on the table surface and to understand the objects' shapes and sizes. For example, a graphic image may be projected directly next to, under, around or in any appropriate relationship to an object on the display surface. It is therefore important to determine various shape characteristics of objects computed from the IR image of the display surface. This function is not provided by touch-sensitive surfaces or screens, for example, most of which are limited to reasoning about one or more discrete points on the surface that correspond to the user's touch. And, it is not clear that prior art interactive display systems determine a relative distance of an object away from the display surface. “Hover state” refers to whether an object is touching the surface of a display surface of an interactive display system (“not hovering”), or is disposed just above the surface of the table (“hovering”). Hover is used in a number of interfaces (e.g., the TabletPC™) to provide a user with pointing feedback without invoking selection, for example, particularly when the pointing device does not have a button for indicating selection. To recover hover state of an object such as the user's finger, it is important to make use of the diffusing display surface, which only reveals objects that are touching or are very close to the display surface of the table in the image produced by the IR vision-sensing system.
A given object will appear brighter in the IR image produced for the display surface, the closer the object is to the display surface. Thus, it would be desirable to determine the maximum brightness of an object, e.g., through a calibration, so that the brightness of the object's IR reflection can be related to the height of the object above the display surface. It would also be desirable to know precisely when an object changes state from hovering just above the display surface to actually touching the display surface.